Multi-core diagnostics and repair using firmware and spare cores

ABSTRACT

Embodiments of the disclosure are directed to an apparatus that comprises a first core susceptible to an error condition, and a second core configured to perform a diagnostic on the first core to identify a cause of the error condition and an action to remedy the error condition in order to recover the first core.

BACKGROUND

The present disclosure relates generally to core diagnostics and repair,and more specifically, to error identification and recovery.

As the number of cores (e.g., processor cores) implemented in a platformor system increases, it may be desirable to provide or facilitate corerecovery. For example, as the number of cores increases, all otherthings being equal it becomes statistically more likely that at leastone core will incur an error. Core recovery may enhance reliability byensuring the availability of operative cores.

In order to provide for core recovery, it is necessary to determinewhether a core subject to an error can be repaired. Current techniquesare unable to determine the cause of the error.

SUMMARY

According to one or more embodiments of the present disclosure, anapparatus comprises a first core susceptible to an error condition, anda second core configured to perform a diagnostic on the first core toidentify a cause of the error condition and an action to remedy theerror condition in order to recover the first core.

According to one or more embodiments of the present disclosure, a methodcomprises performing, by a second core, a diagnostic on a first coreassociated with an error condition, identifying a cause of the errorcondition, identifying an action to remedy the error condition based onthe identified cause of the error condition, applying the identifiedaction, and recovering the first core based on having applied theaction.

According to one or more embodiments of the present disclosure, a systemcomprises a plurality of cores comprising a first core and a firstspare, and firmware configured to select a diagnostic to be applied tothe first core responsive to a detection of an error associated with thefirst core and to provide the diagnostic to the first spare, the firstspare configured to perform the diagnostic on the first core and to passa result of the diagnostic to the firmware.

According to one or more embodiments of the present disclosure, anon-transitory computer program product comprising a computer readablestorage medium having computer readable program code stored thereonthat, when executed by a computer, performs a method comprisingperforming, by a second core, a diagnostic on a first core associatedwith an error condition, identifying a cause of the error condition,identifying an action to remedy the error condition based on theidentified cause of the error condition, applying the identified action,and recovering the first core based on having applied the action.

Additional features and advantages are realized through the techniquesof the present disclosure. Other embodiments and aspects of thedisclosure are described in detail herein. For a better understanding ofthe disclosure with the advantages and the features, refer to thedescription and to the drawings.

BRIEF DESCRIPTION OF THE SEVERAL VIEWS OF THE DRAWINGS

The subject matter which is regarded as the invention is particularlypointed out and distinctly claimed in the claims at the conclusion ofthe specification. The foregoing and other features and advantages ofthe disclosure are apparent from the following detailed descriptiontaken in conjunction with the accompanying drawings in which:

FIG. 1 is a schematic block diagram illustrating an exemplary systemarchitecture in accordance with one or more aspects of this disclosure.

FIG. 2 is a schematic block diagram illustrating an exemplaryenvironment in accordance with one or more aspects of this disclosure.

FIG. 3 is a schematic block diagram illustrating an exemplaryenvironment in accordance with one or more aspects of this disclosure.

FIG. 4 is a flow diagram illustrating an exemplary method in accordancewith one or more aspects of this disclosure.

FIG. 5 is a table illustrating exemplary symptoms, causes, andcorrective actions in accordance with one or more aspects of thisdisclosure.

DETAILED DESCRIPTION

In accordance with various aspects of the disclosure, a core subject toan error (e.g., a failure) may have diagnostics applied to it. Thediagnostics may identify the cause of the error and recommend acondition to run the core in. In some embodiments, the recommendedcondition may be different from a prior run state or condition.

It is noted that various connections are set forth between elements inthe following description and in the drawings (the contents of which areincluded in this disclosure by way of reference). It is noted that theseconnections in general and, unless specified otherwise, may be direct orindirect and that this specification is not intended to be limiting inthis respect.

Referring to FIG. 1, an exemplary system architecture 100 is shown. Thearchitecture 100 is shown as including one or more cores, such as cores102 a-102 f. The cores 102 a-102 f may be organized at any level ofabstraction. For example, the cores 102 a-102 f may be associated withone or more units, chips, platforms, systems, nodes, etc. In some of theillustrative examples discussed below, the cores 102 a-102 f aredescribed as being associated with a processor (e.g., a microprocessor).

One or more of the cores 102 a-102 f may include, or be associated with,one or more memories. The memories may store data and/or instructions.The instructions, when executed, may cause the cores to perform one ormore methodological acts, such as the methodological acts describedherein.

In a multi-core processor, one or more of the cores may be treated as aspare. For example, in connection with the architecture 100, the cores102 a, 102 b, 102 c, and 102 e may generally be operative and the cores102 d and 102 f may be treated as spares under normal or regularoperating conditions. In some embodiments, the cores 102 d and 102 fwhen operating as backup or spare cores may be at least partiallypowered down or turned off to conserve power and/or to extend theiroperational life.

In some embodiments, a core may be susceptible to one or more errors.For example, a core may initially be acceptable (e.g., may befabricated, assembled, or constructed so as to function properly), butmay develop an error at a later point in time.

In some instances, a core may be subject to an error. For example, inconnection with the architecture 100, the core 102 e is illustrated withan ‘X’ through it, thereby indicating that an error may have beendetected in connection with the core 102 e. The error may have beendetected by one or more entities, such as firmware (FW) 104 and/or acore repair engine 106.

The error detected in connection with the core 102 e may be recoverablein the sense that a recovery process may allow the core 102 e to berestored to an operative state or condition, such as a fully orpartially operative state or condition. A recovery process may reset orrestore the core 102 e to a last known good architectural state,optionally based on one or more checkpoints. A recoverable error may be“healed” after recovery if the error is of a transient nature.

The error detected in connection with the core 102 e may benon-recoverable. A non-recoverable error may mean that the core 102 e isidentified as having one or more hardware defects. A non-recoverableerror may be “spareable” or “non-spareable.” In the case of a spareableerror, the core 102 e may be isolated and a spare core (e.g., the core102 d) may be used in place of the core 102 e. In the case of anon-spareable error, another core might not be able to be used in placeof the core 102 e. For example, the non-spareable error may be such thatthe error impacts the operation of the other cores (e.g., the core 102d).

Once an error is detected with the core 102 e, the core 102 e may beisolated and a spare core (e.g., the core 102 d) may assume thefunctionality of the core 102 e. The FW 104 may call or invoke one ormore diagnostic routines 108 in an effort to diagnose and/or recover thecore 102 e. A diagnostic routine 108 may run at any level ofabstraction, such as at a unit level, a memory or cache level, a buslevel, etc. The selection of a diagnostic may be a function of theoperations performed by the core 102 e, code executing on the core 102e, an identification of one or more inputs to the core 102 e when theerror was detected, the state of the core 102 e when the error wasdetected, the state of the other cores 102 a, 102 b, and/or 102 c, orany other condition.

Once a diagnostic is selected by the FW 104, the FW 104 may convey orpass the diagnostic to a spare core, such as the core 102 f. In thismanner, the core 102 f may be treated as, or turned into, a serviceassisted process (SAP) core. An SAP core may perform diagnosis and/orrecovery of a core as described further below. In some embodiments, anSAP core (e.g., the core 102 f) may be the only core to interact orcommunicate with a core (e.g., the core 102 e) that is in an error stateor condition. In some embodiments, an SAP core may select a diagnosticto run or execute.

The selected diagnostic may be run or executed against the core 102 e.The core 102 f, operating as an SAP core, may collect or aggregate theresults of having run the diagnostic against the core 102 e. The core102 f may communicate or pass the results to the core repair engine 106,which may include or be associated with a pervasive infrastructure thathas communication ports to all of the cores 102 a-102 f, or a subsetthereof. The core 102 f and/or the core repair engine 106 may create areport based on the results of the diagnostic.

The results and/or the report may be provided to the FW 104. The resultsand/or the report may be stored in a database 110. The results and/orreport may be provided to a debug and recovery team 112, optionally byway of one or more alerts, alarms, messages, etc. The debug and recoveryteam 112, which may include service personnel, may examine the resultsand/or report to determine one or more actions to take. For example, theactions may comprise one or more of the following: a circuit level fixthat goes in as part of the FW 104, core related parameters(voltage/frequency) update etc. (or) alternatively, the FW 104 candirect the core repair engine 106 with suitable control action(s) to beperformed that may facilitate repairing or recovering the core 102 e.The commands or directives may be communicated from the FW 104 to thecore 102 e via the core repair engine 106.

The architecture 100 may be used to ensure so-called reliability,availability, and serviceability (RAS) performance. For example, it maybe desirable to ensure operability of, e.g., a processor in accordancewith one or more parameters, such as a time standard, a power budget,etc. Diagnosis and recovery of a core (e.g., the core 102 e) mayfacilitate meeting or adhering to RAS standards or metrics.

FIG. 2 illustrates a schematic block diagram of an exemplary environment200 in accordance with one or more aspects of this disclosure. Theenvironment 200 is shown in FIG. 2 as including a processor 202. Theprocessor 202 may include, or be associated with, one or more componentsor devices, such as the cores 102 a-102 f.

The processor 202 may be coupled to one or more entities, such asfirmware (FW) 204. In some embodiments, the FW 204 may correspond to theFW 104 of FIG. 1. The FW 204 may be coupled to a hypervisor (HYP) 206,such as a power hypervisor. The HYP 206 may perform any number offunctions, such as controlling time slicing of operations or routinesassociated with the cores 102 a-102 f, managing interrupts (e.g.,hardware interrupts), re-allocating resources across one or more systemsor platforms, and dispatching workloads.

The environment 200 is shown in FIG. 2 with a number of operations 1-6schematically overlaid on top of it. The operations 1-6 are describedbelow.

In operation 1, one or more cores may be designated as being availablefor RAS purposes. For example, the cores 102 d and 102 f may bedesignated as spare cores.

In operation 2, an error may be detected with a core, such as the core102 e. Recovery of the core 102 e may be possible if, for example, thedetected error is transient in nature.

If recovery is not possible (potentially after one or more iterations ofa recovery process), a so-called “hot fail” may be declared in operation3. The FW 204 may message the HYP 206 to diagnostics-mark (D-mark) orflag the core 102 e, optionally in connection with a processor or systemconfiguration.

In operation 4, the FW 204 or the HYP 206 may D-mark the core 102 e. ByD-marking the core 102 e, the core 102 e might not be accessible toother cores (e.g., the cores 102 a-102 c) for normal operation. The core102 e may only be accessible for diagnosis once it is D-marked.

In operation 5, a spare core (e.g., the core 1020 may be activated orallocated to assume the functionality of the D-marked core (e.g., thecore 102 e). In this manner, the error associated with the core 102 emay be transparent to external devices or entities coupled to, orassociated with, the processor 202. In other words, any potentialperformance degradation resulting from the error may be less than somethreshold or minimized.

In operation 6, a second spare core (e.g., the core 102 d) may be usedas an SAP core to perform a diagnosis on the D-marked core (e.g., thecore 102 e). Once the diagnosis is complete, if recovery was possiblethe D-marking may be removed from the core 102 e, the core 102 e may beput back into service, and the cores 102 d and 102 f may be returned toinactive or spare status. In some embodiments, upon recovery, the core102 e may be treated as a spare and the core 102 f may continue to beutilized as an active core. If recovery of the core 102 e was notpossible, a report may be prepared and recorded regarding, e.g., theinability to recover the core 102 e, diagnostic(s) run against the core102 e, the results of the diagnostic(s), etc.

FIG. 3 illustrates a schematic block diagram of an exemplary environment300 in accordance with one or more aspects of this disclosure. Theenvironment 300 is shown in FIG. 3 as including, or being associatedwith, the cores 102 a-102 f, an interconnect bus 302 (e.g., a powerbus), and an interface unit 304 (e.g., an alter display unit). In someembodiments, the interface unit 304 may be associated with one or morecomponents or devices of a pervasive architecture, such as the corerepair engine 106 of FIG. 1.

The environment 300 is shown in FIG. 3 with a number of operations 1-4schematically overlaid on top of it. The operations 1-4 are describedbelow.

In operation 1, one or more spare cores may be identified for RASpurposes. For example, the cores 102 a and 102 b may be identified asspares.

In operation 2, a core (e.g., the core 102 e) may experience an error.The core 102 e may be fenced off from the rest of the cores and removedfrom the configuration.

In operation 3, a spare core (e.g., the core 102 a) may be used to atleast temporarily replace the fenced core (e.g., the core 102 e).

In operation 4, a second spare core (e.g., the core 102 b) may engage ina diagnosis of the fenced core (e.g., the core 102 e). As part of thediagnosis, the second spare core may scrub the fenced core via theinterface unit 304.

In the environment 300, all communications between the cores 102 a-102 fmay be routed through the interface unit 304. In this manner, a SAP core(the core 102 b in the example described above in connection with FIG.3) may communicate to a core with an error (the core 102 e in theexample described above in connection with FIG. 3) through the interfaceunit 304. The interface unit 304 may include a first interface, such asa pervasive bus communication interface (e.g., a system centeroperations manager (SCOM) interface), to couple to the cores 102 a-102f. The interface unit 304 may couple to the bus 302 via a secondinterface.

FIG. 4 illustrates a method that may be used to repair a core that hasan error associated with it, in accordance with an exemplary embodiment.

In block 402, an error associated with a core may be detected. As partof block 402, the core with the error may be isolated via, e.g., aD-marking or a fence.

In block 404, failure symptoms and/or a cause of weakness may bedetermined responsive to having detected the error of block 402. Forexample, FIG. 5 illustrates, in tabular form, exemplary symptoms 502that may be experienced by a core that has an error, potential causes504 for those symptoms 502, and one or more corrective actions 506 thatmay be engaged to remedy the error or cause 504. For example, referringto FIG. 5, if a circuit or core experiences an issue leading to afunctional failure, such a failure may be caused by (an improper)voltage guard banding; to remedy such a condition, a bump or adjustmentin an applied voltage may be needed. The symptoms 502, causes 504,and/or actions 506 may be determined by one or more entities, such asthe code repair engine 106 of FIG. 1, based on an execution of one ormore diagnostics.

In block 406, FW (e.g., the FW 104 or the FW 204) may direct the corerepair engine 106 to deliver a suitable control action based on thedetermination of block 404. For example, the FW may direct the corerepair engine 106 to command the core experiencing an error to take oneor more of the actions 506 based on detected symptoms 502 and/ordetermined causes 504.

In block 408, the core that experienced an error may be monitored todetermine if the action of block 406 remedied the error condition. Ifthe monitoring of block 408 indicates that the error has been remediedor eliminated, the core that experienced the error may be recovered fornormal use or may be treated as a spare. In this regard, a D-marking orfencing associated with the recovered core may be removed. If themonitoring of block 408 indicates that the error has not been remediedor eliminated, additional diagnostics may be executed and/or a message,warning, or alert may be generated.

In block 410, the FW may be updated to reflect the status of themonitoring of block 408. For example, if the core that experienced theerror condition was recovered, such a status may be denoted by the FW.As part of block 410, the core repair engine 106 may be turned off ordisabled.

It will be appreciated that the events or blocks of FIG. 4 areillustrative in nature. In some embodiments, one or more of the events(or a portion thereof) may be optional. In some embodiments, one or moreadditional events not shown may be included. In some embodiments, theevents may execute in an order or sequence different from what is shownin FIG. 4.

Aspects of the disclosure may be implemented independent of a specificinstruction set (e.g., CPU instruction set architecture), operatingsystem, or programming language. Aspects of the disclosure may beimplemented at any level of computing abstraction.

In some embodiments, a spare core may be used to perform diagnostics ona defective core. A repair engine (e.g., a hardware repair engine),which may optionally be part of a pervasive infrastructure, may attemptto recover the defective core by taking one or more actions, such ascontrolling analog knobs, invoking recovery circuits, optimizingpolicies (e.g., energy, thermal, frequency, voltage, current, power,throughput, signal timing policies), etc.

In some embodiments, a repair engine may interlock with firmware, ahypervisor, an alter display unit, or another entity of a pervasiveinfrastructure to perform bus or system center operations manager drivendiagnostics. The diagnostics may be performed during run time,optionally by D-marking or fencing a core that experiences an error.

In some embodiments, a root cause of an error may be identified. Theerror may be remedied at run time or offline with the assistance of arecovery team, optionally based on data obtained via diagnostics. Insome embodiments, a reboot of a core or processor may occur, such asafter a repair has been performed. In some embodiments, a reboot mightnot be performed following diagnosis, repair, reallocation orde-allocation of a core or processor.

In some embodiments various functions or acts may take place at a givenlocation and/or in connection with the operation of one or moreapparatuses or systems. In some embodiments, a portion of a givenfunction or act may be performed at a first device or location, and theremainder of the function or act may be performed at one or moreadditional devices or locations.

In some embodiments a repair engine (e.g., a core repair engine) may bepart of a pervasive infrastructure of a chip. System operation (e.g.,mainline system operations) may be interleaved with core diagnosticsand/or repair action. Such interleaving may provide for concurrency,such as concurrency in diagnostics.

As will be appreciated by one skilled in the art, aspects of thisdisclosure may be embodied as a system, method or computer programproduct. Accordingly, aspects of the present disclosure make take theform of an entirely hardware embodiment, an entirely software embodiment(including firmware, resident software, micro-code, etc.) or embodimentscombining software and hardware aspects that may all generally bereferred to herein as a “circuit,” “module” or “system.” Furthermore,aspects of the disclosure may take the form of a computer programproduct embodied in one or more computer readable medium(s) havingcomputer readable program code embodied thereon.

Any combination of one or more computer readable medium(s) may beutilized. The computer readable medium may be a computer readable signalmedium or a computer readable storage medium. A computer readablestorage medium may be, for example, but not limited to, an electronic,magnetic, optical, electromagnetic, infrared, or semiconductor system,apparatus, or device, or any suitable combination of the foregoing. Morespecific example (a non-exhaustive list) of the computer readablestorage medium would include the following: an electrical connectionhaving one or more wires, a portable computer diskette, a hard disk, arandom access memory (RAM), a read-only memory (ROM), an erasableprogrammable read-only memory (EPROM or Flash memory), an optical fiber,a portable compact disc read-only memory (CD-ROM), an optical storagedevice, a magnetic storage device, or any suitable combination of theforegoing. In the context of this document, a computer readable storagemedium may be any tangible medium that can contain, or store a programfor use by or in connection with an instruction execution system,apparatus, or device.

A computer readable signal medium may include a propagated data signalwith computer readable program code embodied therein, for example, inbaseband or as part of a carrier wave. Such a propagated signal may takeany of a variety of forms, including, but not limited to,electro-magnetic, optical, or any suitable combination thereof. Acomputer readable signal medium may be any computer readable medium thatis not a computer readable storage medium and that can communicate,propagate, or transport a program for use by or in connection with aninstruction execution system, apparatus, or device.

Program code embodied on a computer readable medium may be transmittedusing any appropriate medium, including but not limited to wireless,wireline, optical fiber cable, RF, etc., or any suitable combination ofthe foregoing.

Computer program code for carrying out operations for aspects of thepresent disclosure may be written in any combination of one or moreprogramming language, including an object oriented programming languagesuch as Java, Smalltalk, C++ or the like and conventional proceduralprogramming language, such as the “C” programming language or similarprogramming languages. The program code may execute entirely on theuser's computer, partly on the user's computer, as a stand-alonesoftware package, partly on the user's computer and partly on a remotecomputer or entirely on the remote computer or server. In the latterscenario, the remote computer may be connected to the user's computerthrough any type of network, including a local area network (LAN) or awide area network (WAN), or the connection may be made to an externalcomputer (for example, through the Internet using an Internet ServiceProvider).

In some embodiments, an apparatus or system may comprise at least oneprocessor, and memory storing instructions that, when executed by the atleast one processor, cause the apparatus or system to perform one ormore methodological acts as described herein. In some embodiments, thememory may store data, such as one or more data structures, metadata,etc.

Embodiments of the disclosure may be tied to particular machines. Forexample, in some embodiments diagnostics may be run by a first device(e.g., a spare core) against a second device (e.g., a core) thatexperiences an error. The diagnostics may be executed during run time ofa platform or a system, such that the system might not be brought downor turned off. In some embodiments, the second device that experiencesthe error may be recovered based on an identification of a cause of theerror and a corrective action applied to the second device to remedy theerror.

The terminology used herein is for the purpose of describing particularembodiments only and is not intended to be limiting of the disclosure.As used herein, the singular forms “a”, “an” and “the” are intended toinclude the plural forms as well, unless the context clearly indicatesotherwise. It will be further understood that the terms “comprises”and/or “comprising,” when used in this specification, specify thepresence of stated features, integers, steps, operations, elements,and/or components, but do not preclude the presence or addition of oneor more other features, integers, steps, operations, element components,and/or groups thereof.

The corresponding structures, materials, acts, and equivalents of allmeans or step plus function elements in the claims below are intended toinclude any structure, material, or act for performing the function incombination with other claimed elements as specifically claimed. Thedescription of the present disclosure has been presented for purposes ofillustration and description, but is not intended to be exhaustive orlimited to the form disclosed. Many modifications and variations will beapparent to those of ordinary skill in the art without departing fromthe scope and spirit of the disclosure. The embodiments were chosen anddescribed in order to best explain the principles of the disclosure andthe practical application, and to enable others of ordinary skill in theart to understand the disclosure for various embodiments with variousmodifications as are suited to the particular use contemplated.

The diagrams depicted herein are illustrative. There may be manyvariations to the diagram or the steps (or operations) described thereinwithout departing from the spirit of the disclosure. For instance, thesteps may be performed in a differing order or steps may be added,deleted or modified. All of these variations are considered a part ofthe disclosure.

It will be understood that those skilled in the art, both now and in thefuture, may make various improvements and enhancements which fall withinthe scope of the claims which follow.

What is claimed is:
 1. An apparatus comprising: a first core susceptibleto an error condition; a second core configured to perform, in responseto a detection of the error condition: a diagnostic on the first core bydirectly communicating with the first core to identify a cause of theerror condition, and an action to remedy the error condition to recoverthe first core; a third core configured to operate as the first corewhen the first core is associated with the error condition; and a corerepair engine, which is separate from the first, second, and thirdcores, configured to facilitate repairing or recovering the errorcondition associated with the first core by communicating directives tothe second core, wherein the third core is further configured to operateas the first core and the first core is configured to operate as a sparesubsequent to the recovery of the first core via the action.
 2. Theapparatus of claim 1, wherein the diagnostic is configured to beselected based on an identification of at least one symptom associatedwith the error condition.
 3. The apparatus of claim 1, wherein thesecond core is configured to perform the diagnostic on the first coreduring a run time of the apparatus.
 4. The apparatus of claim 1, whereinthe third core is configured as a spare and the first core is configuredto continue operation subsequent to the recovery of the first core viathe action.
 5. The apparatus of claim 1, wherein the apparatus is amulti-core processor, and wherein the first core is a failed core, thesecond core is a diagnostics core, and the third core is a spare core,each core being one of a plurality of cores within the multi-coreprocessor.
 6. A system comprising: a plurality of cores comprising afirst core, a first spare, and a second spare; firmware configured toselect a diagnostic to be applied to the first core responsive to adetection of an error associated with the first core and to provide thediagnostic to the first spare; the first spare configured to perform thediagnostic on the first core by directly communicating with the firstcore and to pass a result of the diagnostic to the firmware; the secondspare configured to operate as the first core when the first core isassociated with the error condition; and a core repair engine, which isseparate from the first and second cores, configured to facilitaterepairing or recovering the error condition associated with the firstcore by communicating directives to the second spare, wherein the secondspare is further configured to operate as the first core and the firstcore is configured to operate as a spare subsequent to the recovery ofthe first core via the action.
 7. The system of claim 6, wherein thefirmware is configured to cause the result to be stored in a databaseand to generate at least one of a message, an alert, and a warningregarding at least one of the error and the result.
 8. The system ofclaim 6, wherein the first spare is associated with a first node, andwherein the second spare is associated with a second node.
 9. The systemof claim 6, wherein the first spare and the second spare are associatedwith a processor.
 10. The system of claim 6, further comprising: anengine configured to apply a repair to the first core in order torecover the first core.
 11. The system of claim 10, wherein the appliedrepair is selected based on at least one symptom and conditionidentified by the diagnostic.
 12. The system of claim 10, wherein theengine is configured to receive the repair from the firmware.
 13. Thesystem of claim 10, wherein the repair comprises optimizing a policyselected from at least one of an energy policy, a thermal policy, afrequency policy, a voltage policy, a current policy, a power policy, athroughput policy, and a signal timing policy.
 14. The system of claim10, wherein the repair comprises controlling an analog knob.
 15. Anon-transitory computer program product comprising a computer readablestorage medium having computer readable program code stored thereonthat, when executed by a computer, performs a method comprising:performing a diagnostic on a first core associated with an errorcondition by a second core in direct communication with the first core;operating a third core as the first core when the first core isassociated with the error condition; facilitating a repairing or arecovering the error condition associated with the first core bycommunicating directives to the second core via a core repair engine,which is separate from the first, second, and third cores; identifying acause of the error condition; identifying an action to remedy the errorcondition based on the identified cause of the error condition; applyingthe identified action; recovering the first core based on having appliedthe action; and operating the third core as the first core and the firstcore as a spare subsequent to the recovering of the first core based onhaving applied the action.
 16. The computer program product of claim 15,wherein the method further comprises: prior to recovering the firstcore, isolating the first core to preclude a third core from accessingthe first core.